LAZY TRAINING: INTERACTIVE CLASSIFICATION LEARNING by

LAZY TRAINING: INTERACTIVE CLASSIFICATION LEARNING Michael Rimer Department of Computer Science Master of Science Backpropagation, similar to most learning algorithms that can form complex decision surfaces, is prone to overfitting. This work presents the paradigm of Interactive Training (IT), a logical extension to backpropagation training of artificial neural networks that employs interaction among multiple output nodes. IT methods allow output nodes to learn together to form more complex systems while not restraining their individual ability to specialize. Lazy training, an implementation of IT, is presented here as a novel objective function to be used in learning classification problems. It seeks to directly minimize classification error by backpropagating error only on misclassified samples from outputs that are responsible for the error. Lazy training discourages overfitting and is conducive to higher accuracy in classification problems than optimizing current objective functions, such as sum-squared-error (SSE) and cross-entropy (CE). Experiments on a large, real world OCR data set have shown lazy training to significantly reduce generalization error over an optimized backpropagation network minimizing SSE or CE from 2.14% and 1.90%, respectively, to 0.89%. Comparable results are achieved over eight data sets from the UC Irvine Machine Learning Database Repository, with an average increase in accuracy from 90.7% and 91.3% using optimized SSE and CE networks, respectively, to 92.1% for lazy training performing 10-fold stratified cross-validation. Analysis indicates that lazy training performs a fundamentally different search of the feature space than backpropagation optimizing SSE or CE and produces radically different solutions. These results are supported by the theoretical and conceptual progression from algorithmic to interactive training models.